Optimization Approaches for Learning with Low-rank Regularization

Low-rank modeling has a lot of important applications in machine learning, computer vision and social network analysis. As direct rank minimization is NP hard, many alternative choices have been proposed. In this survey, we first introduce optimization approaches for two popular methods on rank minimization, i.e., nuclear norm regularization and rank constraint. Nuclear norm is the tightest convex envelope of rank function, therefore low-rank optimization using nuclear norm regularization is a convex problem where many convex optimization approaches can be applied. When rank constraint is used, the resulting optimization problems become simpler but are generally non-convex. Thus, algorithms working for these problems are lack of global optimal and may suffer from slow convergence. Except above two common approaches, adaptive non-convex regularizers have recently been proposed, which can better fit singular values. The key idea behind these regularizers is that large singular values are more informative, and thus should be less penalized. The optimization problems are neither smooth nor convex, thus are harder than with nuclear norm regularization or rank constraint. Several algorithms are developed recently which can be applied on these problems, and they are introduced in this survey. Helpful remarks are given for algorithms working within same type of regularizer; and then experiments are performed with both synthetic and real data sets to compare above three different types of regularizers. Finally, we discuss some possible research issues.